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Abstract 

This study investigated social studies achievement as a result of utilizing a nudtimedia-based 
American history software program (Ignite Early American History, 2003) to augment 
textbook and lecture materials for seventh-grade middle school history students in an ethni- 
cally and linguistically diverse urban school district. The instructional software used was 
an interactive midtimedia program designed to teach middle school students through video, 
song, animation, text, and other media to develop critical thinking skills while acqsiiring 
knowledge ofreqsiired content strands ( Ignite Learning, 2003). Teacher and student activities, 
pretest and posttest scores, and instructional methods for experimental and control condi- 
tions were documented in order to provide a comprehensive understanding of the results. 
(Keywords: multimedia instruction, instructional technology, educational software, social 
studies achievement.) 

INTRODUCTION AND PURPOSE OF THE STUDY 

While there is a significant body of literature that discusses technology 
integration in schools and classrooms, there remains a dearth of data-based 
research specifically addressing the issue of the effectiveness of different types 
of educational software in relation to student achievement outcomes (Crosier, 
Cobb, & Wilson, 2002; Mills, 2001; Williams, Boone, & Kingsley, 2004). The 
National Research Council (2002) and others (Campbell, 1969; Cook, 2001) 
found repeatedly that although most educational software is commercially pro- 
duced, “Those with commercial interests are not expected by educators, policy 
makers or the public to use research to support what they sell” (National Re- 
search Council, 2002, p. 96). Consequently, the National Research Council 
(NRC) explained, “Educators are unlikely to draw on scientific knowledge to 
improve their practices in any meaningful way” (p. 96). The current culture 
of high-stakes testing in the United States, with its focus on high expectations 
and accountability, has sparked renewed interest in identifying effective edu- 
cational interventions, including computer assisted instruction, that increase 
student achievement. This investigation adds to the small body of studies uti- 
lizing a rigorous, scientifically based research methodology, as defined by the 
Institute of Education Sciences (IES), to examine student outcomes as a result 
of a technology intervention within a school setting (Poggi, 2003). 
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Interactive Media Focus 

The term multimedia describes any system that combines two or more media 
into a single product or presentation, such as a software program or a Web page. 
Although interactive multimedia capabilities are constantly evolving and have 
become very popular among educators in recent years, the body of research on 
interactive multimedia as an instructional approach is not yet extensive (Alessi 
& Trollip, 2001; Lockard & Abrams, 2004). According to Mayer (2003), a 
multimedia instructional message is “a presentation consisting of words and 
pictures that is designed to foster meaningful learning. Thus, there are two parts 
to the definition: (a) the presentation contains words and pictures, and (b) the 
presentation is designed to foster meaningful learning” (p. 128). 

Mayer (2003) and others (Brouwer, Muller, & Rietdijk, 2007; Thompson, 
2007) have emphasized the unique contributions multimedia brings to the 
learning experience. There are data to support the assertion that multimedia 
capabilities are unique because both sensory stimulation and user navigation 
in interactive multimedia (IMM) parallel students’ natural ways of learning 
(Bagui, 1998; Gibbs, Graves, & Bernas, 2001). Roblyer (1999) asserted that 
the multiple channels through which multimedia communicates to the learner 
seem to be the source of its benefits. The sound, images, animation, and in- 
teractivity in electronic books have also been shown to increase motivation 
and comprehension scores as compared to students’ reading of printed texts 
(Greenlee-Moore & Smith, 1996; Labbo, 2002). According to some researchers 
(Becker, 2000; Mayer, 2003; Moreno, 2006), interactive multimedia is one of 
the best technologies to help students learn. Although claims such as this one 
elicit varying responses among scholars and educators, some research appears to 
indicate that IMM can indeed provide learning benefits (Hancock, Knezek, & 
Christensen, 2007). 

Media-focused research. Much multimedia research has focused on the spe- 
cific media employed for instruction. Clark and Mayer (2003) provided a list of 
several research-based principles for instructional multimedia that focused on 
media type. For example, their multimedia principle discussed the use of text 
with accompanying graphics as opposed to text alone. Similarly, their contiguity 
principle looked at the proximity and placement of a graphic to its correspond- 
ing text. And while much of this research is based on design principles that 
focus on cognitive processing, especially cognitive load theory (Briinken, Plass, 
& Leutner, 2003; Mayer, 2005; Moreno & Valdez, 2005; van Merrienboer & 
Sweller, 2005), priority remains centered on the media elements. 

Supportive Resource Focus 

While the instructional modalities of the software used in this study included 
the expected multimedia components, text remained the predominant vehicle 
for instruction. Text was incorporated into alternate contexts such as maps, 
matching problems, text-documented illustrations, document facsimiles, time- 
lines, concurrent text with spoken audio, and karaoke-style song lyrics. 

With electronic text transformations (e.g., timelines, document facsimiles) 
comprising the bulk of the content of the software used in this study, the Clark 
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and Mayer (2003) media-focused theoretical construct prevalent in much pre- 
vious research did not provide a satisfactory framework for consideration of 
this software as it was implemented. Rather, a typology of supportive resources 
developed by Anderson-Inman and Horney (1998, 2007) that describe how 
the process of reading can be made easier or more educational (Horney & 
Anderson-Inman, 1999) provided the framework for a functional matrix of the 
educational resources in the software. 

Several of these text transformations closely resembled well-documented 
instructional strategies (e.g., advance organizers, graphic organizers, visual dis- 
plays such as timelines, and mnemonic devices) from research into adapting 
challenging textbooks for students experiencing difficulty in reading (Boone & 
Higgins, 2005; Higgins & Boone, 2001; Higgins, Boone, & Lovitt, 2002). The 
overwhelming reliance on text for content delivery in this software suggested 
an instructional design relying more on a concept of supported electronic text 
(Anderson-Inman & Horney, 2007) than the traditional media interactions of- 
ten associated with multimedia. 

Typology of resources. A recently revised description of the Anderson-Inman 
and Horney (2007) typology of resources for supported electronic text includ- 
ed 1 1 resource types: presentational, navigational, translational, explanatory, 
illustrative, summarizing, enrichment, instructional, notational, collaborative, 
and evaluational. Anderson-Inman and Horney stated that the advantage of 
a typology that does not “focus on what media is being used to modify or en- 
hance the electronic text, but rather what function the supportive resources 
play in the reading process” (p. 153) is its usefulness for “teachers, students, 
and parents... to think critically about the modifications, enhancements, and 
additions they encounter when selecting or reading electronic versions of as- 
signed texts” (p. 154). 

Supported electronic text. Overlaying this framework on the supportive 
resources from the American history software used in this research, a clearer de- 
sign for instruction emerged. Resources from the Anderson-Inman and Horney 
(2007) typology that matched the resources provided by the software used in 
the intervention included the following: 

1 . Customizable content (Presentational resource) 

2. Links to resources and related documents and media (Navigational 
resource) 

3. Alternate versions of content (Translational resource) 

4. Descriptions and clarifications of content (Explanatory resource) 

5. Visual representations of content (Illustrative resource) 

6. Questions and testing (Evaluational resource) 

Based on an analysis of the software used in this research, in relation to the 
two conceptual frameworks discussed (e.g., media focused vs. supportive re- 
source), a construct of supported electronic text emerged to best describe the 
intervention used in this study. The research, therefore, focused on typical out- 
come measures associated with content area reading and related learning strate- 
gies (Higgins, Boone, & Lovitt, 2002; Readence & Moore, 1992). Indeed, the 
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questioning format utilized in the pretest/posttest phases of this study has been 
emphasized as a formal and codified reading and comprehension assessment for 
almost 100 years (Readence & Moore, 1992). 

Teaching History and Social Studies 

Many of the difficulties students face in content area learning stem from 
mismatches between the teachers’ instructional approaches and students’ strate- 
gies for cognitive intake and processing of the material presented (Boone & 
Higgins, 2007; Chapin & Messick, 1999; Higgins, Boone, & Lovitt, 2002). 
These mismatches, along with disparate levels of literacy development for stu- 
dents within the same grade level, combine to create formidable challenges for 
the teaching of subject areas where student interest is already low. Mounting 
evidence suggests that students generally find history and social studies dull 
and unimportant, that they have difficulty understanding their textbooks, and 
that overall, they remember very little of what they learned (Ciborowski, 2005; 
Stetson & Williams, 2005; White, 1999). In fact, social studies and history are 
rated by middle school students as two of the least favorite subjects in the cur- 
riculum (Higgins, Boone, & Lovitt, 2002; Lounsbury, 1988; Shaughnessy & 
Haladyna, 1985), with only English receiving more negative reviews about the 
teaching of its content. 

Technology-Supported Social Studies Learning 

Educational technology and interactive multimedia play an increasingly vital 
role in efforts to move social studies from the rote memorization of dates and 
information toward a more student-centered, hands-on, authentic learning 
experience (Bitter & Pierson, 2005; Trinkle & Merriman, 2000). And despite 
movements within the discipline to promote student computer use to facili- 
tate reflective inquiry, decision making, and problem solving (Evans, 2004; 
National Council for the Social Studies, 1994), social studies education for 
the most part continues to focus on traditional, teacher-directed, lecture-and- 
textbook-based approaches and activities (Diem, 2000; Friedman & Hicks, 
2006; White, 1999). 

The research base on the effectiveness of technology as an instructional com- 
ponent for teaching social studies is quite limited (Cantu, 2000). Nonetheless, 
there are data indicating that when integrated effectively, multimedia technol- 
ogy can support history and social studies learning by promoting student-cen- 
tered instruction, increasing learner motivation, and extending and deepening 
understandings of historic and civic concepts (Molebash, 2002). Some studies 
have reported modest positive outcomes for several groups of students who used 
computer-adapted tutorial programs for the practice of social studies skills (Tw- 
yman & Tindal, 2006). In two research studies conducted a full decade apart, 
Higgins, Boone, and Lovitt (1996) found that hypermedia study guides result- 
ed in positive gains for ninth grade social studies students with regard to recall, 
comprehension, and attitudes, while Boon et al. (2006) reported similar results 
in their investigation of high school students’ use of technology-enhanced cog- 
nitive organizers. 
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METHOD 

This study investigated social studies achievement as a result of utilizing a 
multimedia-based, American history software program to augment textbook 
and lecture materials for seventh-grade middle school history students. Student 
pretest and posttest scores on a multiple-choice assessment instrument served as 
the primary data source in this quasi-experimental research design. 

The Early American History Software 

The early American history software used in this study was an online middle 
school curriculum designed to help students learn the content and skills speci- 
fied by state and national academic standards in a student-centered, multi- 
media-rich manner appealing to a wide variety of learning styles and interests 
(Ignite Learning, 2003). The software was a type of computer-aided instruction 
(CAI) that blended networked multimedia technologies for content delivery 
with tools to aid teachers in tracking student progress and designing indi- 
vidualized instruction based on the programs assessments. The program itself 
was Web browser-based, but self-contained in that it prevented access to the 
Internet and World Wide Web while the program was running. The software 
contained 15 units that combined multiple modalities to meet the learning ob- 
jectives in each unit. The researchers had no relationship with the software com- 
pany that produced the history program used in the study. A software license 
was purchased by the school district on a one-year trial basis, during which 
the district technology coordinator requested that the researchers evaluate the 
software. The company provided online and on-site technical assistance for the 
district technology coordinator, classroom teachers, and researchers for the du- 
ration of the trial period. 

Research Design 

This study sought to examine the correlation between multimedia software 
use and student outcome scores, specifically whether use of the American his- 
tory software would significantly raise student achievement scores on a crite- 
rion-referenced, standards-based test. Analyses of data from 1 84 student test 
scores utilized descriptive and inferential statistical procedures to interpret the 
outcome-oriented test results. Pretest and posttest scores for students in control 
and experimental groups were compared using a two-tailed t test with unequal 
variance. A two-tailed t test with unequal variance was implemented because it 
was unclear at the time of comparison which direction mean test scores would 
shift and because a two-tailed t test is more sensitive to changes than a one- 
tailed t test. 

The study aligned with the criteria for methodology, data collection, analysis, 
and description for scientifically based research as explicated in the No Child 
Left Behind Act (NCLB, 2002). Professional literature on NCLB’s criteria for 
scientifically based research (Margolin & Buchler, 2004; NCLB, 2002; Poggi, 
2003; U.S. Department of Education, 2005) guided the methodology, with 
Dawson’s (2004) framework providing the overarching criteria for scientifically 
based research, which included: 
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1 . Empirical methods are used to carry out the research, which is conducted 
in a systematic and consistent manner, with keen attention to detail. 

2. Data collection and analysis are rigorously conducted to ensure that the 
data are collected, analyzed, and interpreted correctly. 

3. Measurements or observational methods that provide scientifically valid 
and reliable measurements across many different measurement points and 
observations are used. 

4. The studies employ experimental or quasi-experimental methodology 
to optimize the researchers’ ability to answer the questions under 
investigation. 

5. Enough data and description should be provided so that future research- 
ers can attempt to replicate the findings by conducting a study using the 
same methods and instruments (p. 5). 

Participants 

Students. Subjects were seventh-grade students enrolled in public middle 
schools in a large urban school district in the southwestern United States. Stu- 
dents in eight separate sections of seventh-grade history, taught by four different 
teachers in three different middle schools participated in the study. The experi- 
mental group of students received treatment (i.e., use of the Ignite! program) in 
addition to textbook- and lecture-based instruction for all units of early Ameri- 
can history study. The control group received textbook and lecture instruction 
only but did not use the Ignite! program. During both instructional conditions, 
the same teacher administered textbook- and lecture-based instruction in pre- 
senting the same information to both groups of students. The overall sample 
size was 1 84 pretests and posttests, obtained from an experimental group com- 
prised of 93 students, and a control group comprised of 91 students. 

Teachers. Four female teachers participated in the study. The teachers worked 
at three different middle schools, collectively teaching American history to a to- 
tal of 637 seventh-grade students each day. Each participating teacher taught an 
experimental group (one full class) of students in which the American history 
software was used as an instructional supplement, as well as a control group (a 
different class) in which the software was not used. This ensured that both con- 
trol and experimental group students had the same teacher, helping to reduce 
the chance of sampling bias. The average age of the teachers was 35 years, with 
an average of 9.5 years of teaching experience. Descriptive information about 
the participating teachers (names are pseudonyms) is shown in Table 1 . Table 
2 provides information about the schools participating in the study (names are 
pseudonyms). 

Setting. The study was conducted in a large, rapidly growing school district 
in the Southwest, in which approximately 40.4% of seventh grade students 
qualified to receive free or reduced lunch, 14.92% of seventh graders were non- 
English proficient or had limited English proficiency, and 11.1% received spe- 
cial education services under an individualized education program (IEP). The 
district’s student population was approximately 14% African American, 33.4% 
Hispanic, and 43.9% Caucasian. However, two of the three schools included 
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Table 1: Descriptive Data for Participating Teachers (all are pseudonyms) 


Teacher 

Name 

Middle 

School 


Age 


Years 

Teaching 

Highest 

Degree 

Romero 

Samuels MS 


30 


5 


B.A. 

Gage 

Hawthorne MS 

55 


26 


M.A. 

Smith 

Hawthorne MS 

31 


7 


M.A. 

Brown 

Jackson MS 


24 


0 


B.A. 

Table 2: Middle School Student Demographic Info (all are pseudonyms) 

Teacher 

Middle 

School 

% of LEP 
Students 


% of 1EP 
Students 

% Eligible for Free 
or Reduced Lunch 

Romero 

Samuels 

22.5 


11.2 


61.8 


Gage 

Hawthorne 

17.5 


12.0 


48.6 


Smith 

Hawthorne 

17.5 


12.0 


48.6 


Brown 

Jackson 

6.7 


12.2 


31.7 



in this study had a higher than average rate of seventh graders eligible to receive 
free or reduced lunch: 50.7% and 61.8%, with minority populations of 61.6% 
and 56.2% respectively. The schools that participated in the study were geo- 
graphically distributed throughout the district, with every effort made to select 
equivalent teachers and students for the treatment and control groups who were 
representative of the districts typical student population. 

Eligibility for school participation. In selecting the sites for the study, par- 
ticipation was limited to schools with adequate technology infrastructure, com- 
puter facilities, and interested teachers. This decision was made after reviewing 
findings from a pilot study (Kingsley, 2003) to determine potential difficulties 
and problems associated with implementing the program in schools on a larger 
scale. Results from the pilot study indicated that schools lacking high-speed, 
high-capacity server and networking capabilities were frequently plagued with 
server and work station crashes, software freezes, very slow response time, and / 
or inability for students to run all of the media segments contained in the 
program. As random selection of participating schools was not possible, the 
decision was made to follow Stake’s (2000) heuristic that in some cases, the op- 
portunity to learn from a site should take priority over concern for its typicality 
of an entire population. 

Teacher training. At the start of the school year, all participating teachers 
attended mandatory introductory training that provided an overview of the 
American history software and familiarized the teachers with available content 
and media options. In the training session teachers learned how to construct as- 
signments, select assessments, create new sections for classes using the program, 
create student logins and passwords, and how to locate and use the multimedia 
options. 
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At the start of the school year, the teachers conducted an orientation session 
to show students in the experimental groups how to log in to the program and 
set their passwords and demonstrated the programs content options and navi- 
gational aids. At that time, the students were given teacher-facilitated, hands-on 
time to familiarize themselves with login procedures and with the program’s 
interface, functionality, and media choices prior to commencement of actual 
instruction. Additionally, all students in the experimental and control groups 
received an overview of the history textbook to be used in the course, an outline 
of the seventh grade history curriculum, and a syllabus for the entire history 
course. 

Data Collection 

Procedure. Prior to using the American history software, each participat- 
ing teacher designated one of her classes to be a treatment group, and another, 
similar class as a control group of students. In all cases, the treatment and 
control classes were inclusive, homogenous, general education seventh-grade 
history classes to which students had been assigned independently of teacher or 
researcher oversight or influence. The overall sample size was 1 84 pretests and 
posttests, obtained from an experimental group comprised of 93 students, and a 
control group comprised of 9 1 students. Every effort was made to select groups 
that would be as similar as possible; however, the use of intact classrooms often 
results in Non-Equivalent Group Design (NEGD). NEGD, a common feature 
of social and educational research, is particularly susceptible to internal validity 
and selection maturation threats, which require further statistical analyses (Han- 
cock, Knezek, & Christensen, 2007). Two additional reliability tests for internal 
consistency (e.g., Split-Half Correlation and Cronbach’s Alpha) were conducted 
to support and validate the findings of this study. 

Instructional conditions. The experimental group of students received treat- 
ment (i.e., use of the American history software) in addition to textbook and 
lecture-based instruction for all units of study. The control group received 
textbook and lecture instruction but did not use the American history software. 
During both instructional conditions, the same teacher administered textbook 
and lecture-based instruction in presenting the same information to both 
groups of students. 

With the treatment group students, teachers reserved a minimum of 20% of 
the instructional time, or approximately one day per week, for use of the Amer- 
ican history software. Regular textbook instruction consisted of using either The 
American Journey (Appleby, Brinkley & McPherson, 2003), or The American 
Nation (Davidson, Castillo, & Stoff, 2002). Both district-approved books were 
similar in content, scope, and sequence of information. The textbooks included 
graphic organizers and other visual aids, such as timelines, photographs and il- 
lustrations, and political maps, as well as vocabulary lists, chapter outlines, and 
chapter summaries. Teachers supplemented book-based instruction with online 
and offline auxiliary activities provided by the textbook publishers, as well as 
with their own materials, worksheets, and selected Web sites. No other instruc- 
tional software programs were used for history instruction during the study. 
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As specified by district policy, students had a copy of their history textbook at 
home, and each classroom had another set for student use at school. Students 
were unable to access the American history software from home. In both the ex- 
perimental and control groups, the curriculum requirements were identical and 
were based on the state history standards scope and sequence. 

The procedure for both instructional groups from pretest to posttest condi- 
tions lasted approximately 7 months. Classes consisted of 50-minute block 
periods encompassing daily review, learning objectives, presentation of new 
information, and in some cases independent practice. On days the software was 
used by students in the experimental groups, class sessions consisted of students 
navigating through the assigned lesson in any order and at their own pace, 
provided that they viewed all of the media contained in the assigned module. 
After viewing the media pieces for the assignment, students completed a Topic 
Review: a six-item multiple-choice assessment built into each lesson. Scores 
from the Topic Reviews were not used for the current study; rather, they served 
as an instructional focal point for students while they used the program. In 
each 50-minute class period where the software was used, students were able to 
complete one full lesson and its accompanying Topic Review. Upon completion 
of the early American history portion of the history course, student participants 
were given the 50-item posttest to measure their knowledge and recall of major 
concepts related to the period of American history from 1492 to 1877 (i.e., the 
period of Reconstruction). 

Instrument. Because the quizzes and topic reviews contained in the software 
were closely tied to that specific content, they were not used as a measure of 
achievement for the study. Rather, an independent, criterion-referenced pretest 
was administered to all participating students at the onset of the seventh-grade 
school year. Material on the pretest consisted of knowledge required to master 
the seventh-grade history curriculum as outlined by state standards. The full 
pretest instrument consisted of 50 multiple-choice questions based on the state’s 
scope and sequence history standards. An identical posttest was administered at 
the conclusion of the 7-month instructional period. 

The multiple-choice pretest-posttest instrument included questions drawn 
from a test bank of 4,500 questions accompanying The American Journey (Ap- 
pleby, Brinkley & McPherson, 2003) history textbook, as well as questions 
contributed by several history teachers in the participating middle schools. Be- 
cause multiple-choice tests tend to focus on basic facts and are not always good 
measures of higher level cognitive processing (Becker, 1992), some multiple- 
choice questions on the pretest and posttest were adjusted slightly to facilitate 
problem-solving, decision making, and/or higher order thinking skills related to 
the concepts covered in the history knowledge being tested. The pretest-posttest 
instrument was compiled by three researchers with experience in designing 
and conducting education research and evaluation who were familiar with this 
research project. Reliability checks on the instrument were conducted indepen- 
dently by the test designers, and discrepancies were discussed and assessed to 
obtain 100% agreement. In addition, due to the identification of NEGD in this 
study, two additional post hoc reliability tests for measuring internal consisten- 
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cy were performed on this instrument. The first test, a Split-Half Correlation, 
randomly divides all the instrument items into two sets and measures the corre- 
lation between the total scores for each randomly divided half of the questions, 
which resulted in a high internal validity score (a=.99) that greatly exceeds the 
acceptable standard, a >0.90. The second, Cronbach’s Alpha, which tends to 
be a more stringent test, revealed high reliability for the control (a=.98) and 
experimental groups (a=.99), as well as the comparison of all groups using the 
pre-test analysis (a=.98). The instrument’s concurrent validity with questions 
from the test bank of questions drawn from the district-approved textbook The 
American Journey was checked and obtained a high validity coefficient (.87). 

The instrument was examined and approved by the district technology coordi- 
nator, the district social studies coordinator, and two of the most experienced 
participating history teachers for content validity to ensure high correlation 
with the scope and sequence of American history content as specified in the 
state curriculum standards. It was then pilot tested with a small sample of doc- 
toral students before the study began. 

Use of the American History Software 

Throughout this study, qualitative data were collected to document the degree 
of fidelity with which the history software was used by all participating teachers 
and students. Weekly classroom observations were augmented by conversations 
and informal interviews with the teachers throughout the 7-month investiga- 
tion period. Transcripts from observations and teacher interviews revealed that 
each of the participating teachers used the program for the equivalent of one 
class period per week throughout the period of investigation. 

Students used the history software in a computer lab where they had access 
to their own computers and were free to work through the assigned modules at 
their own pace. On occasion, the history classes were usurped by another group 
of students who needed to use the computer lab. In these cases the teachers re- 
quested an extra day the following week in order to recoup the missed lab time. 
On a few occasions when the computer lab was occupied on their assigned 
lab day, the teachers used the American history software as a teacher-directed, 
whole-class instructional tool. In these rare cases, the teacher led a discussion of 
the materials and showed the media pieces to the students on a large projection 
screen in the class, while students took notes or completed an outline of the 
material covered. However, the vast majority of time spent using the American 
history software was weekly time in the computer lab with students engaged 
one-to-one with the program on a computer, setting their own pace and using 
the media pieces in any order they preferred. 

Data Analysis 

Descriptive and inferential statistics, including mean, standard deviation, and 
two-tailed t tests, were used on the pretest and posttest scores for students in 
the experimental and control groups. Statistical analyses were conducted using 
the Statistical Product and Service Solutions (SPSS) software to determine the 
significance of variables related to the research question. According to Valdez 
(2004), educational researchers, especially those who have conducted meta- 
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analyses, agree that when used appropriately, technology can improve educa- 
tion in the effect-size range of between 0.30 and 0.40 (Kulik, 2002). Cohen 
(1977) classified effect sizes of around 0.2 as small, around 0.5 as moderate, and 
around 0.8 as large (p. 1). In order to obtain a power rating of .80 with an ef- 
fect size of .50 (moderate effect), there needed to be at least 50 students in each 
of the control and treatment groups, assuming use of a two-tailed test with an 
alpha of .025 (Gay & Airasian, 2000). With sample numbers of more than 50 
for each group, it was possible to measure lesser effects. Two-tailed t tests were 
used because it was unknown whether effects from using the American history 
software would be positive or negative. 

RESULTS 

This investigation examined whether there was a statistically significant differ- 
ence between pretest and posttest achievement scores for students who used the 
American history software compared to students who did not use the program. 
Using the computer software program Statistical Product and Service Solutions 
(SPSS), descriptive and inferential statistics were compiled from the pretest and 
posttest scores of students in control and experimental groups. The mean scores 
of control and experimental groups on pretests and posttests were calculated, 
then compared using a two-tailed t test with unequal variance. 

Pretest. For students in the pretest control group (n= 91), the average num- 
ber of correct answers was 33.60 out of 50 total questions with a standard de- 
viation of 5.30, while the average number of correct answers for all students in 
the pretest experimental group (n=93) was 30.95 out of 50 total questions, with 
a standard deviation of 6.12. In other words, students in the control group had 
a 67.2% pretest average for correct answers, while students in the experimental 
group had a pretest average of 61.9% for correct answers. 

Posttest. At the end of the instructional period being studied, the average 
number of correct answers for students in the posttest control group (n=91) 
was 36.66 out of a total of 50 questions with a standard deviation of 5.58, the 
equivalent of 73.32% correct, while the average number of correct answers for 
students in the posttest experimental group (n=93) was 37.04 of 50 total ques- 
tions with a standard deviation of 5.51, the equivalent of 74.07% correct. 

The mean posttest scores indicated that students who used American history 
software, as well as those who did not use it, both increased their test scores 
from pretest to posttest conditions. These data describe the classic “selection- 
maturation threat” potential of any NEGD project, in which improvement was 
observed in both the “non-equal” experimental and control groups. However, 
examination of the percentage increase between pretest control and pretest ex- 
perimental groups to posttest control and posttest experimental group revealed 
that students in the control group increased their mean test scores an average of 
6.1%, while students in the experimental group increased their mean test scores 
an average of 12.2%, or approximately twice as much. This difference in mean 
test scores was statistically significant. Moreover, although the reliability mea- 
sures (i.e., Split-Half, Cronbach’s Alpha) demonstrated reliability for all groups 
for the pre-test — post-test reliability was significantly reduced (a=0.48), provid- 
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ing further evidence that suggests the experimental treatment was sufficient to 
significantly alter the outcomes measured by this instrument. 

The significance level associated with the difference in test score results be- 
tween the control and experimental groups was less than 0.01%. The question 
less confidently answered was the likelihood of attribution of the treatment to 
the 12.2% mean test score increase for students in the experimental group ver- 
sus the 6.1% mean test score increase for students in the control group. 

DISCUSSION 

This section addresses the research question: Was there a significant differ- 
ence between pretest and posttest achievement scores for students who used 
the American history software as compared to students who did not use the 
program? Results indicated statistically significant positive effects on overall 
achievement scores for students who used the American history software. Mean 
test scores for students who used the software improved by 12.2% and an aver- 
age of 6.09 more correct answers from pretest to posttest, while mean scores 
for control group students improved by 6.1%, an average of 3.06 more correct 
answers from pretest to posttest. The significance level in a statistical study is 
the risk associated with not being 100% confident that what was observed in 
an experiment or quasi-experiment was due to the treatment or what was be- 
ing tested. In this case the treatment was student usage of the American history 
software program. Since the impact of all other potential factors on the differ- 
ences observed between outcomes of treatment and control groups cannot be 
eliminated confidently, some level of probability (i.e., the p value) is assigned 
and reported. On a two-tailed t test of unequal variance, a very high level of 
significance was found, /<=0. 0000000337623, where p represents the probability 
that the increase in mean test scores was attributable to something other than 
use of the American history software. The data suggest that the difference in 
outcome scores for students in the experimental group was likely due to their 
use of the American history software. 

In addition to reporting outcomes and probability levels for errors, studies 
conducted in consonance with the No Child Left Behind (2002) definition for 
scientifically based research must also report the effect size and statistical power 
of a study. Statistical power is related to the variance: the smaller the variation 
relative to each group (e.g., between the experimental and control groups), the 
larger a sample size must be in order to obtain a high power rating. The power 
of a statistical hypothesis test measures the test’s ability to reject the null hy- 
pothesis when it is actually false — that is, to make a correct decision (Hinkle, 
Wiersma, & Jurs, 1998). Obviously, the higher the power rating, the more reli- 
able the statistical test. The maximum power a test can have is 1, and the mini- 
mum is 0. Ideally, researchers would strive to have a high power or a number 
close to 1. In this study, for the control group of students the power was 0.965, 
and for the experimental group the power was 1.00. In other words, there was 
a 96.5% statistical likelihood that the two-tailed t test was able to detect the ef- 
fects for the control group of students, and a 100% chance that it was able to 
detect the effects for the experimental group. Consequently, it can be asserted 
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that the results of the two-tailed t test on both the control and experimental 
groups yielded valid, reliable results. Overall, results of this quasi-experiment 
suggest a strong link between use of a technology-enhanced intervention and 
higher outcome achievement scores for this group of middle school learners. 

Limitations of the Study 

Because participants in this study were not randomly selected but were in- 
stead part of a cohort, the generalizability of the results to similar student popu- 
lations is considered lower than if the sampling process had been completely 
random. The non-experimental group design (NEGD) of this study precludes 
the drawing of causal inferences. Generalizability may also be limited by the 
fact that all of the participating teachers were female. 

Implications and Future Research 

This study was designed and conducted in consonance with criteria speci- 
fied in the No Child Left Behind Act's (2002) definition of scientifically based 
research. With instructional technology playing an increasingly central role in 
all academic areas, more research, and more effective approaches are needed to 
document student achievement related to computer-based training and edu- 
cational programs (Bull et ah, 2005; U.S. Department of Education, 2005). 

This study adds to the body of scientifically based research literature on student 
achievement directly linked to the use of educational software. Bull et al. de- 
scribe the compelling need for this sort of research by stating “ [t] o date there 
have been no documented systemic increases in student achievement and learn- 
ing directly attributable to technological innovation” (p. 218). They add, 

“[tjhere is no area in which well-conceived and effectively implemented research 
could be of greater value than in the area of [educational] technological in- 
novation” (p. 218). The current study responds to calls for accountability from 
scholars, policymakers, and educators at all levels for rigorous evidence indicat- 
ing whether technology investments can truly support student learning (Jones 
et ah, 2004-2005) in educational settings. Furthermore, this study adds to the 
very limited body of research on the effectiveness of technology as a component 
for teaching social studies (Cantu, 2000; Diem, 2000). 

Results of this study suggest several potential directions for further explora- 
tion. One possible avenue for investigation addresses concerns that a major 
shortcoming of scholarly literature on the efficacy of technology in education 
is that the research varies tremendously in methodology, sampling, and focus. 
Researchers (Kirkpatrick & Cuban, 1998; Waxman, Lin & Michko, 2003) 
have documented studies with large variations in sampling, such as differences 
in student grade levels, socioeconomic classes, and aptitudes. The current study 
employed a quasi-experimental methodology that implemented all NCLB cri- 
teria for scientifically based research, including a disparate sampling of teachers 
and students from the school district, and a strong emphasis on using the in- 
structional intervention with fidelity. An experimental research design employ- 
ing in-depth qualitative and quantitative data analyses would provide greater 
insight into the causal factors surrounding the differences in student achieve - 

Journal of Research on Technology in Education 215 

Copyright © 2006, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 



ment, while also providing information related to the processes teachers use to 
integrate new technology into their existing curriculum. 

In the current study, the data suggest that use of a software program affected 
student achievement scores on a standards-based, multiple-choice test; however, 
many questions about the effects of educational software on student learning 
remain unanswered. Another possible direction for further research would be 
to investigate gains in student achievement if the program were to be used with 
students for more than the 20% of instructional time implemented in this study. 
For instance, students who did not use the software showed an average mean test 
score increase of around 6%, while those who used the program had mean test 
score increases of about 12%, or twice as much. It would be interesting to inves- 
tigate what might happen if software usage were increased to 25%, or even to 
50% of instructional time. Would students continue to show gains on standard- 
ized assessments, or would a point of diminishing returns be reached? 

Final directions for further investigation include exploring whether test score 
increases attributed to use of the history software would be enough to truly 
make a difference in whether students pass their seventh grade history courses 
(i.e., practical significance vs. statistical significance) and whether knowledge 
gained from use of the software transfers to more complex learning tasks outside 
the context of a standardized written examination. 
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